Systems and methods for machine learning approaches to management of healthcare populations

ABSTRACT

A method for providing treatment recommendations for a patient to a physician is disclosed. The method includes receiving health information associated with the patient, determining a first risk score for the patient based on the health information using a trained predictor model, determining a second risk score for the patient based on the health information and at least one artificially closed care gap included in the health information using the predictor model, determining a predicted risk reduction score based on the first risk score and the second risk score, determining a patient classification based on the predicted risk reduction score, and outputting a report based on at least one of the first risk score, the second risk score, or the predicted risk reduction score.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. provisionalapplication 62/936,374, filed Nov. 15, 2019.

BACKGROUND OF THE DISCLOSURE

The present invention relates to systems and methods for analysis andmanagement of heart failure populations. Heart failure (HF) has alifetime prevalence of 1 in 3 in the United States and is responsiblefor approximately 1 million hospital discharges annually and 1 in 8deaths in the United States. The estimated annual cost of HF in theUnited States is $30.7 billion and that amount is expected to more thandouble to $69.7 billion by 2030, costing every United States citizen anaverage of $244 annually.

In response to these rising costs, new models of healthcare andreimbursement are being developed. In these “value-based care” models,management of many chronic conditions like heart failure is extendingbeyond singular patient-physician encounters to instead treat disease ata population scale. The general goal of such models is to improvepatient outcomes while reducing/containing costs by delivering care thatkeeps patients efficiently managed and reduces the frequency of highcost/high acuity encounters. Optimizing this kind of management at apopulation level requires an effective means to identify and stratifypatients in need of intervention and, ideally, identify appropriateinterventions to deploy. At present, there is a critical lack ofvalidated, data-driven models to support these population health goals.

Data science approaches, including machine learning, are well-suited toassist with these tasks. For example, one of the first papers on thissubject in 1995 showed that a neural network could utilizeechocardiography data to predict 1-year mortality in 95 heart failurepatients with accuracy that was superior to a linear model or clinicaljudgement. Since then, numerous additional studies with thousands ofpatients have shown significant promise for machine learning to predicthospitalization, readmission, or death in patients with heart failure.

Previously published models using machine learning for risk predictionsin patients with heart failure have two primary limitations with regardto their utility in optimizing clinical population health management.First, most models have used small, systematically collected andannotated datasets (e.g., as from a clinical trial) or focused on animportant, but narrow, clinical setting (e.g., in-hospital mortalityduring heart failure hospitalization for acute decompensation). Suchapproaches, while valid and appropriate within their respectiveconstraints, are not necessarily generalizable to a broad andheterogeneous heart failure population, as characterized in “real world”clinical data. The second limitation is that none of the publishedfindings using machine learning models have led to clinically-relevant,actionable results.

Thus, what is needed is a system for providing clinically-relevant,actionable treatment recommendations for patients who should be but arenot receiving evidence-based care generalizable to a broad andheterogeneous heart failure population.

BRIEF SUMMARY OF THE DISCLOSURE

The present disclosure includes systems and methods for machine learningapproaches to management of heart failure populations. Morespecifically, the present disclosure provides systems and methods forproviding clinically-relevant, actionable treatment recommendations forpatients who should be but are not receiving evidence-based caregeneralizable to a broad and heterogeneous heart failure population. Thepresent disclosure provides systems and method for generating a list ofpatients rank ordered by highest estimated benefit of providingadditional treatments and/or other resources such as medication in orderto more efficiently provide resources to patients.

Some embodiments of the present disclosure provide a method forproviding treatment recommendations for a patient to a physician. Themethod includes receiving health information associated with thepatient, determining a first risk score for the patient based on thehealth information using a trained predictor model, determining a secondrisk score for the patient based on the health information and at leastone artificially closed care gap included in the health informationusing the predictor model, determining a predicted risk reduction scorebased on the first risk score and the second risk score, determining apatient classification based on the predicted risk reduction score, andoutputting a report based on at least one of the first risk score, thesecond risk score, or the predicted risk reduction score.

To the accomplishment of the foregoing and related ends, the invention,then, comprises the features hereinafter fully described. The followingdescription and drawings set forth in detail certain illustrativeaspects of the invention. However, these aspects are indicative of but afew of the various ways in which the principles of the invention can beemployed. Other aspects, advantages and novel features of the inventionwill become apparent from the following detailed description of theinvention when considered in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow for training models for predicting 1-year all-causemortality using electronic health record (EHR) data and predictingmortality risk with and without artificially closing care gaps.

FIG. 2 is a graph of the number of patients for each gap for which thegap was open/untreated or closed/treated.

FIG. 3A is a graph of mean area-under-curve for linear and non-linearmodels.

FIG. 3B is a graph of area under curve for years 2010-2018 for thelinear and non-linear models.

FIG. 4 is a graph of the top twenty ranking variables using XGBoost.

FIG. 5A is a graph of average mortality rate corresponding to risk scorebin data derived from training data across all training years.

FIG. 5B is a graph of a distribution of predicted risk score in aprediction set (alive patients), which was then translated to predictedmortality rate using the relationship shown in FIG. 5A.

FIG. 6A is a scatter plot of risk score and corresponding benefit forindividual patients in a prediction set.

FIG. 6B is a graph of average mortality rate before and after care gapclosure simulation in selected groups.

FIG. 7 is a graph of estimated lives saved by various stratificationtechniques during simulation of care gap closure using XGBoost.

FIG. 8 is an exemplary process for predicting all-cause mortality inpatients with heart failure for a predetermined time period (i.e., oneyear), as well as providing treatment recommendations for a patient to aphysician.

FIG. 9 is an exemplary system for implementing the process of FIG. 8.

While the invention is susceptible to various modifications andalternative forms, specific embodiments thereof have been shown by wayof example in the drawings and are herein described in detail. It shouldbe understood, however, that the description herein of specificembodiments is not intended to limit the invention to the particularforms disclosed, but on the contrary, the intention is to cover allmodifications, equivalents, and alternatives falling within the spiritand scope of the invention as defined by the appended claims.

DETAILED DESCRIPTION OF THE DISCLOSURE

The various aspects of the subject invention are now described withreference to the annexed drawings. It should be understood, however,that the drawings and detailed description hereafter relating theretoare not intended to limit the claimed subject matter to the particularform disclosed. Rather, the intention is to cover all modifications,equivalents, and alternatives falling within the spirit and scope of theclaimed subject matter.

As used herein, the terms “component,” “system” and the like areintended to refer to a computer-related entity, either hardware, acombination of hardware and software, software, or software inexecution. For example, a component may be, but is not limited to being,a process running on a processor, a processor, an object, an executable,a thread of execution, a program, and/or a computer. By way ofillustration, both an application running on a computer and the computercan be a component. One or more components may reside within a processand/or thread of execution and a component may be localized on onecomputer and/or distributed between two or more computers or processors.

The word “exemplary” is used herein to mean serving as an example,instance, or illustration. Any aspect or design described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs.

Hereafter, unless indicated otherwise, the following terms and phraseswill be used in this disclosure as described. The term “provider” willbe used to refer to an entity that operates the overall system disclosedherein and, in most cases, will include a company or other entity thatruns servers and maintains databases and that employs people with manydifferent skill sets required to construct, maintain and adapt thedisclosed system to accommodate new data types, new medical andtreatment insights, and other needs. Exemplary provider employees mayinclude researchers, clinical trial designers, oncologists,neurologists, psychiatrists, data scientists, and many other personswith specialized skill sets.

The term “physician” will be used to refer generally to any health careprovider including but not limited to a primary care physician, amedical specialist, an oncologist, a neurologist, a nurse, and a medicalassistant, among others.

The term “researcher” will be used to refer generally to any person thatperforms research including but not limited to a radiologist, a datascientist, or other health care provider. One person may be both aphysician and a researcher while others may simply operate in one ofthose capacities.

Furthermore, the disclosed subject matter may be implemented as asystem, method, apparatus, or article of manufacture using programmingand/or engineering techniques to produce software, firmware, hardware,or any combination thereof to control a computer or processor baseddevice to implement aspects detailed herein. The term “article ofmanufacture” (or alternatively, “computer program product”) as usedherein is intended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. For example, computerreadable media can include but are not limited to magnetic storagedevices (such as hard disk, floppy disk, magnetic strips), optical disks(such as compact disk (CD), digital versatile disk (DVD)), smart cards,and flash memory devices (such as card, stick). Additionally it shouldbe appreciated that a carrier wave can be employed to carrycomputer-readable electronic data such as those used in transmitting andreceiving electronic mail or in accessing a network such as the Internetor a local area network (LAN). Transitory computer-readable media(carrier wave and signal based) should be considered separately fromnon-transitory computer-readable media such as those described above. Ofcourse, those skilled in the art will recognize many modifications maybe made to this configuration without departing from the scope or spiritof the claimed subject matter.

In this disclosure, ARB refers to angiotensin II receptor blocker, ACEIrefers to active angiotensin-converting enzyme inhibitor, ARA refers toaldosterone receptor antagonist, ARNI refers to angiotensinreceptor-neprilysin inhibitor, AUC refers to area under the receiveroperating characteristic curve, EBBB refers to evidence-basedbeta-blocker, ECG refers to electrocardiogram, and EHR refers toelectronic health record.

The inventors leveraged a large 20-year retrospective dataset derivedfrom a health system (Geisinger) that was an early adopter of electronichealth record (EHR) technology to develop a predictive model for allpatients with heart failure using machine learning. This model includeda comprehensive set of input variables, including 6 “care gap”indicators. A “Care Gap” is defined as the discrepancy betweenrecommended best practices and the care that is actually provided.

Importantly, this novel incorporation of evidence-based care gaps into apredictive model represents a methodology for driving clinical actionfrom a machine learning model (not just predicting risk but predictingreduction in risk, or “benefit”, as a result of action). Moreover, it isdemonstrated how such insights might be utilized through populationhealth management efforts to simultaneously stratify risk andtherapeutic benefit at an individual patient level to efficiently deployhealthcare resources.

Methods

EHR Data Collection

Patients with heart failure over a 19 year period (January 2001-February2019) were identified from Geisinger EHRs. Heart failure was definedusing the validated eMERGE phenotype. All clinical encounters since 6months prior to the heart failure diagnosis date, including outpatientoffice visits, hospital admissions, emergency room visits, lab tests andcardiac diagnostic studies (e.g. echocardiograms or electrocardiograms),were identified as independent samples.

Model Inputs

FIG. 1 is a flow for training models for predicting 1-year all-causemortality using EHR data and predicting mortality risk with and withoutsimulating care gap closure/treatment by artificially closing care gaps.1-year all-cause mortality was studied in a large cohort of heartfailure patients using machine learning models to integrate clinicalvariables, measures from diagnostic studies (e.g. echocardiography andelectrocardiography) and evidence-based care gap variables fromelectronic health records. Mean area under the ROC curve (AUC) from a‘split-by-year’ training scheme was reported to evaluate modelperformance. The best performing model was then used to estimate riskreduction (potential benefit) by artificially closing care gaps in aprospective prediction set and to evaluate the efficiency ofbenefit-driven patient prioritization. FIG. 1 identifies variousexemplary machine learning models that may be used as part of thisprocess, including logistic regression (“LR”), random forest (“RF), andXGBoost. Within FIG. 1, the abbreviation BP indicates blood pressure.

A total of 80 variables were collected from the EHR (see FIG. 1): 8clinical variables (age, sex, height, weight, smoking status, heartrate, systolic and diastolic blood pressures), use of loop diuretics, 12biomarkers (hemoglobin, eGFR, CKMB, lymphocytes, HDL, LDL, uric acid,sodium, potassium, NT-proBNP, troponin T, A1c), 44 non-redundantechocardiographic variables, 9 ECG measurements (such as QRS duration)and 6 care gap variables (described below). Lab values, vital signs andECG measures closest to the encounter date within a 6-month window wereextracted. All echocardiographic measurements recorded in the Xceleradatabase within 12 months of the encounter date were extracted. If nomeasurements were available within the specified time window, thevariable was set to missing. EHR data preprocessing and cleaning isfurther detailed in the “EHR Data Preprocessing” section below. It isunderstood that these variables are just one of many possiblecollections of variables that could be used to train similar models.Moreover, additional data types such as medical image data, medicalsignals data (e.g. electrocardiograms), genomic data, etc., can be usedas inputs to the model.

EHR Data Preprocessing

Physiologic limits for echocardiographic variables were defined withassistance from a cardiologist with expertise in echocardiography. Datacleaning included removal of 1) redundant variables that were deriveddirectly from other variables and 2) values outside physiologicallypossible ranges as defined by a cardiologist, including physiologicallyimpossible values likely due to human error (e.g. LVEF <0% or >100%,height and weight below 0). The removed values were then set as missing.

Since the predictive models require complete datasets, missing data forcontinuous variables were imputed using two steps. First, missing valuesin between encounters for an individual patient were linearlyinterpolated if complete values were found in the adjacent encounters.Next, measurements that were missing in 90% or more samples werediscarded to ensure enough samples are available for imputation for eachmeasurement, and the remaining missing values were imputed using arobust Multivariate Imputation by Chained Equations (MICE).

Missing values for diastolic function (represented as a categoricalvariable), were imputed by training a One-vs-All logistic regressionclassifier from all samples where diastolic function was available.Diastolic function was reported as an ordinal variable based on level ofabnormality, with −1 for normal, 0 for abnormal (no grade reported), and1, 2 or 3 for grades I, II and III diastolic dysfunction, respectively.

Care Gap Variables

Six evidence-based, actionable interventions (care gap variables) wereintroduced to the machine learning models to study their associationwith patient outcomes: 1) flu vaccine administration, 2) hemoglobin A1cin goal (<8%), 3) BP in goal (blood pressure <140/90 mmHg), 4) activeevidence-based beta-blocker (EBBB), 5) active angiotensin-convertingenzyme inhibitor (ACEI), angiotensin II receptor blocker (ARB) orangiotensin receptor-neprilysin inhibitor (ARNI) and 6) activealdosterone receptor antagonist (ARA). These care gap variables weredefined with assistance from a cardiologist and a pharmacist with heartfailure expertise. Detailed inclusion/exclusion criteria are listed inTable 1 below. A blinded chart review validation of each care gapvariable is detailed in the “Care Gap Validation” section below. It isunderstood that there are treatments or interventions other than thelisted care gap variables that can be input to the model, for examplemedications, clinic visits, provider visits to the patient home, etc.Note that for new therapies or medications for which outcomes have notyet been acquired in a large retrospective clinical dataset tofacilitate the most accurate machine learning model training, datashowing the effect of a therapy on a particular outcome of interest canbe used until enough data is captured to generate a new model.

TABLE 1 Care Gap Definitions Care Gap Inclusion Exclusion Gap ClosureFlu vaccine N/A Allergy Received flu vaccine in the current flu seasonBlood pressure N/A N/A Open (not in goal) if >= 2 (BP) in goal of the 5most recent readings in the past 12 months are >140 for systolic or >90for diastolic A1c in goal Diagnosis of diabetes N/A Most recent A1cwithin (defined using problem the last 6 months <8% list diagnoses)Evidence-based Diagnosis of heart failure Bradycardia (heart rate <60 byaveraging Currently taking EBBB beta-blocker (EBBB) with most recentleft up to 5 most recent readings in last 6 months) ventricular ejectionOn inotropic therapy fraction (LVEF) <40% History of 2^(nd) or 3^(rd)degree heart block without ICD or pacemaker Hypotension (systolicpressure <100 mmHg by averaging last 5 readings in past 6 months) Severechronic obstructive pulmonary disease (COPD) or asthma Allergy orcontraindications Active Diagnosis of heart Pregnancy Currently takingangiotensin-converting failure with most History of angioedema ACEI orARB or ARNI enzyme inhibitor recent LVEF <40% Hypotension(ACEI)/Angiotensin II receptor blocker (ARB)/Angiotensin Serumcreatinine >2 in any of preceding receptor-neprilysin 3 measurementsinhibitor (ARNI) Potassium >5 in any of previous 3 measurements Allergyor contraindications Newly initiated dialysis Aldosterone Diagnosis ofheart Hypotension Currently taking ARA receptor failure with most Serumcreatinine >2 in any of preceding antagonist (ARA) recent LVEF <35% 3measurements Potassium >5 in any of preceding 3 measurements On dialysisAllergy or contraindicationsCare Gap Validation

To validate the accuracy of the defined care gap variables, tworeviewers independently and manually reviewed 50-100 charts for eachcare gap variable in blinded fashion. Specifically, a questionnaire wascreated in REDCap for each care gap variable, with questions coveringpatient inclusion (e.g. if patient has heart failure), gap open/closedstatus (e.g., if patient's most recent A1C was <8%), and exclusioncriteria (e.g., if patient is allergic to flu vaccine). 50-100 caseswere randomly selected for each care gap from our database whilebalancing positive and negative cases for each criterion. For example,for flu vaccine, 25 cases had an expected open gap (no flu vaccinereceived) and 25 with an expected closed gap (flu vaccine received). Thenumber of cases was determined based on how many criteria/questions wereincluded for each care gap. Note that since there were multipleexclusion criteria involved for the medication related gaps with rarefrequency in the EHR, we did not balance the cases based on exclusioncriteria, but only ensured that representative cases were included. Forthe selected cases, patients' medical record number (MRN, uniqueidentifier) and encounter dates were provided to the reviewer. Thereviewer then filled out the questionnaire by reviewing the patient'schart in EPIC, using the provided encounter dates as the reference date.This was used as the ground truth to compare with our calculated caregap data. A summary of the review results is presented in Table 2 below.In Table 2, N/A means there are no inclusion/exclusion restrictions forthe gap.

TABLE 2 Cases Accuracy (%) Reviewed (N) Inclusion Open/Closed ExclusionACEI/ARB/ARNI 100 93 98 99 Aldosterone 100 94 99 97 receptor antagonistBP in goal 50 N/A 98 N/A A1c in goal 50 90 100 N/A Evidence-based 100 8898 94 beta-blocker Flu vaccine 50 N/A 100 100 Primary Outcome

Machine learning models were used to predict all-cause mortality 1 yearafter the associated encounter date. Survival duration was calculatedfrom the date of death (cross-referenced with national death indexdatabases on a monthly basis) or last known living encounter from theEHR. It is understood that this is an example of a single clinicallyrelevant endpoint, however, additional endpoints include but are notlimited to hospital admissions, emergency department or clinic visits,total cost of care, adverse outcomes such as stroke or heart attack,etc.

Machine Learning Model Training and Evaluation

First, a linear logistic regression classifier was used for itssimplicity (particularly for examining directionality of associationsbetween model inputs and the primary outcome), and then compared to theperformance to non-linear models including random forest and XGBoost (ascalable gradient tree boosting system). These nonlinear models werehypothesized to improve predictive accuracy by capturing more complex,non-linear relationships among input variables. The best performingmodel was selected for subsequent analysis of care gap closure effectestimation. Models were evaluated using a ‘split-by-year’ form ofcross-validation as described in the “Machine Learning Model Evaluation”section below.

Machine Learning Model Evaluation

The most recent encounters were excluded in all alive patients withheart failure (as of Feb. 9, 2019) as a prospective, prediction dataset(a clinically “actionable” dataset). All remaining samples (encounters)with known outcome status were used for model evaluation.

To evaluate the proposed model, the inventors deviated from thetraditional cross-validation approach, because the random split approachmisrepresents the “real-world” deployment scenario. Instead a‘split-by-year’ procedure was followed to divide the samples intotraining (past) and test sets (future). To deploy a model, the model istrained on all available data prior to the present date and applied tothe patient's most recent encounter, therefore, one can retrospectivelyevaluate the model as if it were deployed in a given date. For each year(e.g. 2010), the cutoff date was set as January 1^(st) of that year(Jan. 1, 2010) such that all encounters prior to the cutoff date wereused for training, and the first encounter for a given patient after thecutoff (but within the calendar year, from Jan. 1, 2010-Dec. 31, 2010)was used for testing. This process was repeated for years 2010-2018.

Area under the receiver operating characteristic (ROC) curve (AUC) fromthe test set was obtained and overall model performance was reported asthe mean AUC and standard deviation over all training years. The averageimportance and ranking for each individual variable over all trainingyears was obtained to identify the most important variables. The opensource python packages “scikit-learn” (version 0.20.0) and “xgboost”(version 0.80) were used to implement the machine learning pipeline andevaluate the models.

After the training stage, an optimal set of hyper-parameters wasobtained, and further used to re-train the entire dataset to obtain afinal model. The final model was then used on the held out actionableprediction dataset (most recent encounters from all patients alive as ofFeb. 9, 2019) to obtain a likelihood score for each individual patient.This likelihood score, which is referred as the risk score, ranged from0 to 1, with higher values corresponding to higher risk of mortality.

During training, a risk score was obtained for each individual sample inthe test set. These risk scores were binned into 20 groups of 0.05increments from 0-1, and the true mortality rate was calculated usingground truth from samples within that group. The average event rate overall training years for a specific bin was used to estimate the eventrate as a function of the computed risk scores in the prediction set.This enabled a mapping of risk scores to the mortality event rate.

Benefit Prediction in Alive Patients by Simulation of Care Gap Closure

To study the effect of closing care gaps on improving patient outcomes,care gaps were artificially closed (i.e. changing the value from1=open/untreated to 0=closed/treated) while keeping all other variablesunchanged. A care gap was not closed in patients who met the exclusioncriteria for that care gap (for example, a patient with bradycardia whocould not be treated with EBBB). First, a logistic regression was usedto estimate the associated directionality of each care gap variable withthe predicted mortality risk (e.g. receiving flu vaccine associated withdecreased mortality risk). No care gaps that had a negative orundetermined relationship with the outcome (i.e. BP in goal, asdescribed later) were closed. For care gaps which had a positiverelationship with the outcome, the gap closure was simulated in the bestperforming non-linear model by artificially closing the gap andre-calculating the risk score using the same model.

After the simulation, the change in risk score, i.e., the differencebetween baseline risk score with care gaps open and risk score with caregaps closed, was calculated for each patient, which was furthertranslated into an estimated benefit, i.e. reduction in estimatedmortality rate. The cumulative sum of the benefit from all patients wasthen used to provide an estimated number of lives that could be saved byclosing care gaps. In some embodiments, the risk score with care gapsopen and/or the risk score with care gaps closed can be provided to andused by a physician and/or a provider to estimate the risk of death of aspecific patient. In this way, the physician and/or provider canestimate if the patient has a high likelihood of dying within the year(or other time period) so that appropriate resources such as palliativecare physicians can be provided to the patient at an appropriate time.

Results

Study Population

24,740 patients with heart failure who collectively had 945,404encounters (median age 76 years, 45% female) within the EHRs that fitthe inclusion criteria were identified. Note that while encounters areused as a prediction input to the models in this scenario, theprediction input can be configured differently for example by using“episodes” where multiple encounters are concatenated or otherwisecombined into one point in time from which the prediction is made.Tables 3 and 4 below show summary statistics. On average, each patienthad 38 encounters (interquartile range (IQR): 10-49). The median followup duration was 3.4 years (IQR: 1.4-6.3 years) using reverseKaplan-Meier, and 12,594 (51%) had a recorded death. Data are reportedas median [interquartile range], or percentage.

TABLE 3 Basic Demographics and Patient Characteristics. All Most recentencounter from (N = 945,404 encounters alive patients from 24,740patients) (N = 12,416) Age (yr) 76 [67-83] 75 [65-84] Male (%) 55 53Smoking History 64 62 (current or ever smoking) (%) Height (cm) 168[157-175] 168 [159-175] Weight (kg) 85 [70-102] 86 [72-105] DiastolicPressure (mmHg) 68 [60-74] 70 [61-78] Systolic Pressure (mmHg) 124[112-137] 124 [112-138] Heart Rate (bpm) 72 [64-80] 73 [64-82] EjectionFraction (%) 52 [37-57] 52 [40-57] High-density lipoprotein 45 [36-54]45 [38-52] (HDL) (mg/dL) Low-density lipoprotein 80 [61-101] 83 [64-100](LDL) (mg/dL) N-terminal-pro hormone 3264 [1054-6129] 2960 [869-5443]B-type natriuretic peptide (NT-proBNP) (pg/mL) Troponin T (ng/mL) 0.02[0.01-0.09] 0.03 [0.01-0.14]

TABLE 4 All encounters (N = 945,404) Percentage/Median [IQR] DescriptionAge (years) 76 [67-83] Sex (% male) 55% Smoking status (% smoker) 64%Height (cm) 168 [157-175] Weight (kg) 85 [70-102] Heart rate (bpm) 72[64-80] Diastolic blood pressure 68 [60-74] (mm Hg) Systolic bloodpressure 124 [112-137] (mm Hg) LDL (mg/DL) 80 [61-101] Low-densitylipoprotein HDL (mg/DL) 45 [36-54] High-density lipoprotein A1c (%) 6.4[5.8-7.1] CKMB (ng/mL) 3.4 [2.2-6.1] Creatine kinase-muscle/brainHemoglobin (g/dL) 12.3 [10.9-13.6] Lymphocytes (%) 19 [12-25] Potassium(mmol/L) 4.3 [4.0-4.6] NT-proBNP (pg/mL) 3264 [1054-6129] N-terminal-prohormone B-type natriuretic peptide Sodium (mmol/L) 140 [137-142]Troponin T (ng/mL) 0.02 [0.01-0.09] eGFR (mL/min/1.73{circumflex over( )}m2) 52.5 [37.3-60] Estimated glomerular filtration rate Uric acid(mg/dL) 7.0 [6.4-7.6] Urate in serum or plasma Loop diuretics (% taking)62% QRS duration (ms) 106 [90-138] QT (ms) 418 [382-454] QT interval QTc(ms) 462 [436-492] QT interval corrected for heart rate PR interval (ms)174 [150-204] Vent rate (bpm) 74 [64-86] Ventricular rate RR interval(ms) 814 [694-936] Average RR interval P axis (degree) 53 [38-66] R axis(degree) 10 [−30-56] T axis (degree) 66 [27-104] LVEF(%) 52 [37-57]Physician-reported left ventricular ejection fraction AI dec slope(cm/s2) 219 [204-234] Aortic insufficiency deceleration slope AI max vel(cm/s) 359 [348-369] Aortic insufficiency maximum velocity Ao V2 VTI(cm) 36 [31-42] Velocity-time integral of distal to aortic valve flow AoV2 max (cm/s) 152 [122-191] Maximum velocity of distal to aortic valveflow Ao root diam (cm) 3.2 [3.0-3.5] Aortic root diameter Asc Aorta (cm)3.3 [3.1-3.5] Ascending aorta diameter EDV (MOD*-sp2) (ml) 113 [94-135]LV end-diastolic volume: apical 2-chamber EDV (MOD*-sp4) (ml) 114[94-137] LV end-diastolic volume: apical 4-chamber EDV (sp2-el**) 117[98-140] LV end-diastolic volume: apical 2-chamber EDV (sp4-el**) 118[98-143] LV end-diastolic volume: apical 4-chamber ESV (MOD*-sp2) (ml)61 [45-80] LV end-systolic volume: apical 2-chamber ESV (MOD*-sp4) (ml)63 [46-81] LV end-systolic volume: apical 4-chamber ESV (sp2-el**) (ml)63 [47-83] LV end-systolic volume: apical 2-chamber ESV (sp4-el**) (ml)66 [50-85] LV end-systolic volume: apical 4-chamber IVSd (cm) 1.2[1.0-1.3] IV septum dimension at end-diastole LA dimension (cm) 4.3[3.8-4.8] Left atrium dimension LAV (MOD*-sp2) (ml) 75 [64-85] Leftatrium volume: apical 2-chamber LAV (MOD*-sp4) (ml) 75 [63-85] Leftatrium volume: apical 4-chamber LV V1 VTI (cm) 19 [17-21] Velocity-timeintegral: proximal to the obstruction LV V1 max (cm/s) 90 [76-104]Maximum LV velocity: proximal to the obstruction LVIDd (cm) 5.0[4.4-5.6] LV internal dimension at end-diastole LVIDs (cm) 3.6 [3.0-4.2]LV internal dimension at end-systole LVLd ap2 (cm) 8.1 [7.8-8.6] LVlong-axis length at end diastole: apical 2-chamber LVLd ap4 (cm) 8.1[7.7-8.6] LV long-axis length at end diastole: apical 4-chamber LVLs ap2(cm) 7.2 [6.8-7.7] LV long-axis length at end systole: apical 2-chamberLVLs ap4 (cm) 7.2 [6.8-7.7] LV long-axis length at end systole: apical4-chamber LVOT area (M) (cm2) 3.4 [3.2-3.6] LV outflow tract area LVOTdiam (cm) 2.1 [2.0-2.2] LV outflow tract diameter LVPWd (cm) 1.1[1.0-1.3] LV posterior wall thickness at end-diastole MR max vel (cm/s)482 [466-498] Mitral regurgitation maximum velocity MV A point (cm/s) 79[64-92] A-point maximum velocity of mitral flow MV E point (cm/s) 94[74-115] E-point maximum velocity of mitral flow MV P1/2t max vel (cm/s)115 [100-128] Maximum velocity of mitral valve flow MV dec slope (cm/s2)497 [409-567] Mitral valve deceleration slope MV dec time (s) 0.20[0.17-0.24] Mitral valve deceleration time PA V2 max (cm/s) 95 [85-102]Maximum velocity of distal to pulmonic valve flow PA acc slope (cm/s2)689 [533-821] Pulmonary artery acceleration slope PA acc time (s) 0.10[0.08-0.12] Pulmonary artery acceleration time Pulm. R-R (s) 0.86[0.83-0.90] Pulmonary R-R time interval RAP systole (mm-Hg) 8.0[7.1-8.8] Right atrial end-systolic mean pressure RVDd (cm) 3.5[3.3-3.6] Right ventricle dimension at end-diastole TR max vel (cm/s)275 [248-303] Tricuspid regurgitation maximum velocity Diastolicfunction (severity: %) −1: 12%  −1: normal; 0: 29% 0: abnormal (no gradereported); 1: 31% 1: grade I dysfunction; 2: 17% 2: grade IIdysfunction; 3: 11% 3: grade III dysfunction ACEI/ARB/ARNI (% open)  9%¹ See Table 1 Aldosterone receptor 14% See Table 1 antagonist (%open) BP in goal (% open) 23% See Table 1 A1c in goal (% open) 26% SeeTable 1 Evidence-based beta-blocker  7% See Table 1 (% open) Flu vaccine(% open) 39% See Table 1

Of the 12,146 patients who were alive as of Feb. 2, 2019, 9,474, (78%)had at least one open care gap, and 501 (4%) had 4 or more care gapsopen as of their most recent encounter dates. FIG. 2 is a graph of thenumber of patients for each gap for which the gap was open/untreated orclosed/treated. The sum of those groups represents the number ofpatients who were eligible for the gap (i.e., who fit the inclusioncriteria). Depending on the gap, 20-74% of eligible patients had an opengap. Additional details are available in Table 5 below. In Table 5,percentage of exclusion and percentage of open are calculated based onnumber of included encounters (i.e., encounters during which a patientwas eligible and thus satisfied the inclusion criteria for taking themedicine). In FIG. 2, EBBB (also mentioned in FIG. 1) stands forevidence-based beta-blocker, ACEI stands for activeangiotensin-converting enzyme inhibitor, ARB stands for angiotensin IIreceptor blocker, ARNI stands for angiotensin receptor-neprilysininhibitor, and ARA stands for aldosterone receptor antagonist.

TABLE 5 Training set (N = 784,965) Prediction set (N = 12,146) InclusionExclusion Gap Open Inclusion Exclusion Gap Open ACEI/ARB/ARNI 183,918(23%) 37,508 (20%) 72,943 (40%) 2,991 (25%) 447 (15%) 1,219 (41%)Aldosterone receptor 145,098 (18%) 38,421 (26%) 111,326 (77%) 2,301(19%) 500 (22%) 1,712 (74%) antagonist BP in goal 784,965 (100%) 0 (0%)176,330 (22%) 12,146 (100%) 0 (0%) 2,473 (20%) A1c in goal 372,774 (47%)0 (0%) 201,881 (54%) 5,088 (42%) 0 (0%) 3,010 (59%) Evidence-based beta-183,918 (23%) 9,734 (5%) 183,918 (34%) 2991 (25%) 104 (3%) 780 (26%)blocker Flu vaccine 784,965 (100%) 11,353 (1%) 300,368 (38%) 12,146(100%) 177 (1%) 6,849 (56%)Accuracy for Predicting all-Cause Mortality Using Machine Learning

All three machine learning models predicted 1-year all-cause mortalitywith AUCs above 0.70, and the non-linear models achieved higher averageAUCs (random forest: 0.76±0.02, XGBoost: 0.77±0.03) compared to linearlogistic regression (0.73±0.02; FIG. 3). FIG. 3A is a graph of mean AUCfor linear and non-linear models. Both non-linear models performedbetter than linear logistic regression (LR) at predicting 1-yearall-cause mortality, with XGBoost (XGB) having the highest average AUC.FIG. 3B is a graph of area under curve for years 2010-2018 for linearand non-linear models.

FIG. 4 is a graph of the top twenty ranking variables using XGBoost.Besides commonly used clinical variables (age, weight) and biomarkers(HDL, LDL), echocardiographic variables are highly important forpredicting 1-year all-cause mortality in patients with heart failure.See Table 4 above for variable descriptions. Variable importancerankings using XGBoost demonstrated that 15 of the top 20 variables wereechocardiographic measures. Logistic regression results demonstratedthat 5 of the 6 care gap variables (all but BP in goal) had an expectedpositive association such that an open gap was associated with higherrisk of 1-year all-cause mortality. Only these 5 variables were used topredict the effect of closing care gaps in subsequent models.

Predicting Benefit of Closing Care Gaps

XGBoost was chosen as the final model to predict the benefit of closingcare gaps in the alive patients, since the XGBoost model had the highestAUC in the retrospective data. The distribution of risk scores is shownin FIGS. 5A-B. FIG. 5A is a graph of average mortality ratecorresponding to each risk score bin derived from the training dataacross all training years. FIG. 5B is a graph of a distribution ofpredicted risk score in the prediction set (alive patients), which wasthen translated to predicted mortality rate using the relationship shownin FIG. 5A. The number of encounters included in each training/test foldper year is included in Table 6 below. Of the 12,146 alive patients,based on the estimated mortality rate, 2,662 (21.9%) patients werepredicted to die within 1 year. The drop in the testing set in 2018 isdue to insufficient follow-up duration (<1 year) for alive patients asof the data collection date (Feb. 9, 2019).

TABLE 6 Training Testing Year All Dead Alive All Dead Alive 2010 109,71128,005 81,706 3,841 685 3,156 2011 143,659 35,552 108,107 4,441 9033,538 2012 189,711 46,572 143,139 5,382 1,053 4,329 2013 240,825 58,765182,060 6,632 1,248 5,384 2014 301,471 72,965 228,506 7,670 1,480 6,1902015 375,567 90,717 284,850 8,378 1,481 6,897 2016 459,659 109,493350,166 8,986 1,371 7,615 2017 553,164 128,405 424,759 10,243 1,5428,701 2018 657,322 150,800 506,522 4,653 1,351 3,302

Artificially closing the 5 care gaps that positively associated withmortality resulted in 2,495 (20.5%) patients being predicted to diewithin 1 year. This resulted in a predicted absolute risk reduction of1.4% (range: 0-31%, absolute) in mortality rate, and 167 (6.3% of 2,662)additional patients would be expected to survive beyond 1 year assumingall 5 care gaps could be closed.

The relationship between risk and benefit (risk reduction) was furtherinvestigated by comparing the predicted benefits among severalsubgroups. FIG. 6A is a scatter plot of risk score and correspondingbenefit for individual patients in the prediction set (N=12,146).Negative reductions in mortality rate reflect a detrimental effect ofclosing care gaps on mortality risk as predicted by the non-linearXGBoost model in a small proportion of patients. FIG. 6B is a graph ofaverage mortality rate before and after care gap closure simulation inselected groups. Note that risk is not equivalent to benefit sincepatients at similarly high mortality risk levels do not have the samepredicted benefit of closing care gaps.

FIG. 6B shows that the overall average benefit (“Overall Average”) waspredicted to be relatively small and was primarily driven by the largegroup of patients with low mortality risk at baseline (risk score<0.2)as well as low benefit after closing the care gaps (<5% reduction inmortality rate) (“Low Risk, Low Benefit”). There was, however, asubgroup of patients predicted to be high risk for mortality (riskscore>0.5) who were also predicted to have high benefit after closinggaps (>10% reduction in mortality rate, “High Risk, High Benefit”). Yet,not all high-risk patients were predicted to have high benefit, asevidenced by another subgroup of patients who had similarly high risk atbaseline but minimal risk reduction after closing the care gaps (“HighRisk, Low Benefit”).

Patient Prioritization to Efficiently Close Care Gaps Through PopulationHealth Management

Assuming that a population health management team could be assembled anddeployed to close care gaps, the efficiency of its efforts would dependon effective guidance as to which patients to target first in a rankordered fashion. To demonstrate the potential value of machine learningto optimize care team resource deployment in this setting, the number oflives predicted to be saved versus the number of patients receiving anintervention (in which all eligible gaps were subsequently assumedclosed) was plotted for several different prioritization strategies:

Strategy 1: Random Prioritization

Strategy 2: Randomly prioritizing any patient with at least one opencare gap

Strategy 3: Rank ordering patients by the number of open care gaps

Strategy 4: Stratifying patients using the Seattle Heart Failure riskscore

Strategy 5: Stratifying patients according to the XGBoost model'spredicted “benefit” (i.e. mortality risk reduction)

FIG. 7 is a graph of estimated lives saved by various stratificationtechniques during simulation of care gap closure using XGBoost.Prioritization of patients according to predicted benefit is the mostefficient resource allocation method based on having the highestpredicted patient survival (y-axis) relative to the number of patientsneeded to treat (x-axis). Note that the slopes of the plotted lines areinversely proportional to the number needed to treat and thus steeperlines represent more efficient patient prioritization. The small drop inlives saved at the far right-hand side of the line corresponding to the“Benefit Driven” model reflects the patients in which closing the caregaps had a predicted negative impact on mortality risk, as shown in FIG.6A.

FIG. 7 demonstrates that the proposed machine learning benefitstratification model (strategy 5) was the most efficient. That is,benefit stratification had the steepest slope of any prioritizationstrategy and thus, in a resource constrained environment, maximized thepredicted total number of lives saved for a given number of patientinterventions.

Discussion

Optimized population health management demands novel, data-drivenapproaches for allocating healthcare resources, particularly within newvalue-based care models. This study has made considerable advancestoward the development of such an approach for heart failure thatcombines extensively and carefully curated clinical data and machinelearning. The model incorporates important clinical variables,quantitative measures from common diagnostic studies such asechocardiography and electrocardiograms, as well as evidence-basedinterventions in the form of “care gaps”. The results show that amachine learning model with these inputs can achieve good accuracy topredict 1-year all-cause mortality in patients with heart failure.Furthermore, the explicit representation of clinical care gaps in themodel represents a new paradigm for guiding clinical action with machinelearning. Specifically, the present disclosure shows how these care gapinputs can be used to predict risk reduction associated with specificinterventions on an individual patient level.

These model predictions can provide guidance to integrated healthsystems working to efficiently distribute scarce clinical resources(e.g., care teams) to patients who need them the most. Importantly, mostpublished models and clinical scoring systems rely heavily on riskprediction, which could be used to prioritize distribution of healthcareresources. However, risk is not equivalent to benefit and thus patientswith identical risk of 1-year mortality can have very differentpredicted benefit from interventions. Thus, deployment of resourcesbased simply on risk is unlikely to be efficient, as demonstrated by thesuperiority of the predictive model's predicted performance over theSeattle Heart Failure score for prioritizing patient interventions.

Comparison to Other Predictive Machine Learning Models in Heart Failure

Several studies have been published in recent years using machinelearning to predict outcomes (mostly survival) in patients with heartfailure. These studies used various methods, from traditionalclassification (e.g. logistic regression, random forest) to customdeveloped algorithms (Contrast pattern aided logistic regression withprobabilistic loss function) to predict mortality in heart failure. Thereported accuracies (AUC) vary from 0.61-0.94, while mostly centeredaround 0.75-0.8.

On the surface, the model performance is comparable with these priorstudies. However, several critical differences should be noted, as theyreflect the more challenging prediction task accomplished by thepredictive model presented by this disclosure. First, the model wasdesigned for prospective implementation in a “real world” clinicalsetting, as reflected in both the training/testing scheme and theprospective randomized clinical trial initiated using this model. Hence,the approach relied on clinical EHR data (as opposed to data collectedduring a controlled clinical trial) and allowed for its associatedchallenges (e.g., incomplete and/or erroneous data). Second, mostprevious studies have focused on specific subgroups of heart failure,such as stratifications by preserved) or reduced ejection fraction orpatients with acute decompensation; or focused on prediction in specificsettings, such as in-hospital mortality or mortality followingadmission. Tur analysis focused broadly on all patients with heartfailure and considered both in-patient and out-patient encounters, againreflecting the needs of a continuously updating population healthmanagement approach.

Given this more challenging prediction task, it is noteworthy that themodel performance was in line with previous studies. This achievementwas driven primarily by two attributes of the dataset. Foremost, thesample size of the study is more than an order of magnitude larger(close to 1 million encounters from 24 thousand patients) compared toprevious studies (mostly a few hundreds to thousands), which allows formore generalizable models with reduced chance of overfitting.Additionally, the model included a comprehensive set of patient features(input variables), including data from diagnostic studies such asechocardiograms, which are highly important for predicting all-causemortality in the setting of heart failure (FIG. 4) and a more generalcardiology population (note that the current study contains somepatients from a previous study on 171,510 patients). In contrast, mostprevious studies were limited to basic clinical information(demographics, vital signs), results from lab tests, and co-morbidities.Only one study included additional diagnostic measures fromechocardiography and electrocardiograms and reported an AUC of 0.72 forall-cause mortality despite a small patient sample (n=397), furthersupporting the importance of these quantitative diagnostic data.

Another major drawback of most prior studies is the lack of actionablemodel results which can be used clinically. Therefore, although a largenumber of accurate models have been developed over the last decade topredict outcomes in patients with heart failure, few have truly impactedclinical practice. A recent study attempted to address this issue byevaluating associations between treatments (various medications) andoutcomes among 4 subgroups of heart failure identified usingunsupervised clustering in a retrospective dataset. The authors of thestudy showed marked differences in outcomes and different responses tomedications among the 4 subgroups, which could help to define effectivetreatment strategies specific to each subgroup. In line with that study,this concept was taken one step further and 6 evidence-basedinterventions (care gaps) were introduced into the machine learningmodel and used these variables as actionable “levers” in the model topredict individual patient outcomes after a clinical action. Byartificially closing these care gaps, it is predicted that an additional167 patients could survive longer than 1 year.

Despite the fact that these interventions (care gaps) are recommended innational guidelines based on demonstrated benefit (e.g. even fluvaccination has been associated with decreased all-cause mortality inheart failure), the prevalence of open care gaps remains a significantproblem in medicine. For example, in patients with heart failure,therapies proven to prolong life are used at staggeringly low rates:only 57% are receiving ACE inhibitors, 34% are receiving evidence-basedbeta blockers, and 32% are receiving mineralocorticoid antagonists. Thisproblem is highly complex and unlikely to be solved by relying onindividual providers to change practice. However, new value-based caremodels can likely address this problem more effectively by creatingorganized care teams. These teams will require accurate, reliable datascience, such as that presented in this disclosure, in order tosuccessfully allocate resources.

Surprisingly, the “BP in goal” care gap had a negative relationship withoutcome, in contradiction to the evidence-based guidelines based onobservational studies which have shown that lower blood pressuresassociated with reduced risk of adverse events in heart failure.However, the “blood pressure paradox” has also been noted in multiplestudies where lower blood pressure or pronounced changes in bloodpressure (increases or decreases) was associated with poor outcomes. Inthe current study, the linear logistic regression model demonstrated aninconsistent relationship between blood pressure and survival, i.e.negative association in some training years and positive association inothers, with a small, negative relationship on average (data not shown).In the present disclosure, a machine learning model configured topredict 1-year all-cause mortality with good accuracy in a large cohortof patients with heart failure is presented. The results leveragingnearly 1 million encounters from over 24,000 patients show that thesemodels can be used to not only risk stratify patients, but to alsoefficiently prioritize patients based on predicted benefits ofclinically relevant evidence-based interventions. This approach willlikely prove useful for assisting heart failure population healthmanagement teams within new value-based payment models. It is alsocontemplated that a model configured to predict all-cause mortality fortime periods other than one year, including six months, two years, threeyears, four years, five year, or other appropriate time periods couldalso be generated. Additionally, as described above, additionalclinically relevant endpoints can be used to train the predictivemachine learning model.

Turning now to FIG. 8, an exemplary process 100 for predicting all-causemortality in patients with heart failure for a predetermined time period(i.e., one year), as well as providing treatment recommendations for apatient to a physician is shown. The process 100 predicts risk scoresfor the patient based on a machine learning model trained on clinicalvariables (e.g. demographics and labs), electrocardiogram measurements,electrocardiograph measurements, and evidence-based care gap variablesas described above. The process 100 can be employed in a populationhealth analytics module that is relied on by a care team including thephysician to prioritize patients who should be but are not receivingevidence-based care.

At 102, the process 100 can receive health information associated withthe patient. The health information can include at least a portion of anEHR associated with the patient. The EHR can be stored in a database ofa provider. In some embodiments, the health information can include theeighty variables including eight clinical variables (age, sex, height,weight, smoking status, heart rate, systolic and diastolic bloodpressures), use of loop diuretics, twelve biomarkers (hemoglobin, eGFR,CKMB, lymphocytes, HDL, LDL, uric acid, sodium, potassium, NT-proBNP,troponin T, A1c), forty-four non-redundant echocardiographic variables,nine ECG measurements (such as QRS duration) and the six care gapvariables described above. In some embodiments, the health informationmay not include BP in goal. The process 100 can then proceed to 104.

At 104, the process 100 can determine a first risk score for the patientbased on the health information using a trained predictor model. Thetrained predictor model can be a linear model such as linear logisticregression or a non-linear model such as random forest or XGBoost asdescribed above. The predictor model can be trained to predict riskscores of all-cause mortality for a predetermined time period, such asone year, although it is appreciated that the model could be trained topredict all-cause mortality for other time periods six months, twoyears, three years, four years, five year, or other appropriate timeperiods or other appropriate clinical endpoints. The process 100 canprovide at least a portion of the health information to the model andreceive the first risk score from the model. The first risk score canrepresent a baseline score corresponding to an actual predictedmortality risk of the patient. The process 100 can then proceed to 106.

At 106, the process 100 can determine a second risk score for thepatient based on the health information and at least one artificiallyclosed care gap included in the health information using the predictormodel. The process 100 can artificially close appropriate care gaps bychanging the value of each open care gap from 1=open/untreated to0=closed/treated while keeping all other variables included in thehealth information unchanged. The process 100 may not close certain caregaps in patients who meet the exclusion criteria for that care gap. Forexample, a patient with bradycardia who could not be treated with EBBBwould not have the EBBB care gap closed. The process 100 can thenprovide the health information, which has been modified to close anyappropriate care gaps, to the model and receive the second risk scorefrom the model. The second risk score can represent a simulated scorecorresponding to what the predicted mortality risk of the patient wouldbe if all appropriate open care gaps are closed. For some patients, at106, the process may not be able to close any care gaps, either becausethe care gaps are already closed or cannot be closed for patient whomeet the exclusion criteria for certain care gaps as described above, inwhich case the second risk score will be the same as the first riskscore. The process 100 can then proceed to 108.

At 108, the process 100 can determine a predicted risk reduction scorebased on the first risk score and the second risk score. The process 100can calculate the predicted risk reduction score by determining thedifference between the first risk score and the second risk score. Theprocess 100 can then proceed to 110.

At 110, the process 100 can determine a patient classification based onthe predicted risk reduction score. The process 100 can determine thepatient classification by comparing the predicted risk reduction scoreof the patient against predicted risk reduction scores of a group ofother patients. The group of other patients can include other patientstreated by the provider. The process 100 can determine a rank of thepatient predicted risk reduction score of the patient compared to thegroup of patients (i.e., using strategy 5 described above). For example,the process 100 can determine that the predicted risk reduction score of0.3 is the five hundredth highest predicted risk reduction score out often thousand patients. The process 100 can then proceed to 112.

At 112, the process 100 can generate and output a report based on atleast one of the first risk score, the second risk score, or thepredicted risk reduction score. For example, the report can include theraw first risk score, the raw second risk score, and the raw predictedrisk reduction score. The report can include the raw rank of thepredicted risk reduction score of the patient compared to the group ofpatients (e.g., that the predicted risk reduction score is the fivehundredth highest predicted risk reduction score out of ten thousandpatients) or a percentile rank of the predicted risk reduction score(e.g., that the predicted risk reduction score is in the ninety-fifthpercentile of all patients of the provider). The report can include anyappropriate graphs and/or charts generated based on the first riskscore, the second risk score, and/or the predicted risk reduction score.The report can be displayed to a physician using a display such as acomputer monitor or a screen integral to a tablet computer, smartphone,laptop computer, etc. In some embodiments, the report can be output to astorage device including a memory. In some embodiments, the report caninclude the raw first risk score and the second raw risk score. Thefirst risk score and the second risk score can be used by a physicianand/or a provider to estimate the risk of death of the patient. In thisway, the physician and/or provider can estimate if the patient has ahigh likelihood of dying within the year (or other time period) so thatappropriate resources such as palliative care physicians can be providedto the patient at an appropriate time.

Turning now to FIG. 9, an exemplary system 210 for implementing theaforementioned disclosure is shown. The system 210 may include one ormore computing devices 212 a, 212 b in communication with one another,as well as with a server 214 and one or more databases or other datarepositories 216, e.g., via Internet, intranet, ethernet, LAN, WAN, etc.The computing devices also may be in communication with additionalcomputing devices 212 c, 212 d through a separate network 218. Althoughspecific attention is paid to computing device 212 a, each computingdevice may include a processor 220, one or more computer readable mediumdrive 222, a network interface 224, and one or more I/O interfaces 226.The device 212 a also may include memory 228 including instructionsconfigured to cause the processor to execute an operating system 230 aswell as a population health analytics module 232 for predicting 1-yearall-cause mortality in patients with heart failure as well as providingtreatment recommendations for a patient to a physician as describedherein. The population health analytics module 232 can be used toexecute at least a portion of the process 100 described above inconjunction with FIG. 8.

The methodology described above for driving clinical action based onpredicted reduction in risk (i.e., benefit) can be applied to themanagement of any particular population (other than a heart failurepopulation) in healthcare including but not limited to diabetes,pulmonary disease, renal disease, rheumatologic disorders,musculoskeletal conditions, endocrinopathies, etc. Furthermore, themethodology can be extended to predict risk reduction for any particularclinical outcome of interest, including but not limited to outcomes suchas mortality, additional adverse clinical events such as stroke or heartattack, hospitalization, total cost of care or other healthcareutilization metrics, etc.

Thus, as described herein, the present disclosure provides systems andmethods for providing clinically-relevant, actionable treatmentrecommendations for patients who should be but are not receivingevidence-based care generalizable to a broad and heterogeneous heartfailure population.

While the present disclosure may be susceptible to various modificationsand alternative forms, specific embodiments have been shown by way ofexample in the drawings and have been described in detail herein.However, it should be understood that the present disclosure is notintended to be limited to the particular forms disclosed. Rather, thepresent disclosure is to cover all modifications, equivalents, andalternatives falling within the spirit and scope of the presentdisclosure as defined by the following appended claims.

This written description uses examples to disclose the presentdisclosure, including the best mode, and also to enable any personskilled in the art to practice the present disclosure, including makingand using any devices or systems and performing any incorporatedmethods. The patentable scope of the present disclosure is defined bythe claims and may include other examples that occur to those skilled inthe art. Such other examples are intended to be within the scope of theclaims if they have structural elements that do not differ from theliteral language of the claims, or if they include equivalent structuralelements with insubstantial differences from the literal languages ofthe claims.

Finally, it is expressly contemplated that any of the processes or stepsdescribed herein may be combined, eliminated, or reordered. Accordingly,this description is meant to be taken only by way of example, and not tootherwise limit the scope of this present disclosure.

What is claimed is:
 1. A method for providing treatment recommendationsfor a patient to a physician, the method comprising: receiving healthinformation associated with the patient, the health informationcomprising a plurality of input variables for a trained predictor model,at least some of the plurality of input variables associated with arespective first value reflecting a state thereof, the healthinformation including data derived from one or more of echocardiogram orelectrocardiogram signal data; identifying a plurality of open care gapswithin the health information, each open care gap comprising an inputvariable of the plurality of input variables for which the respectivefirst value corresponds to an open state; determining, using the trainedpredictor model, a first risk score relating to a clinical endpoint ofthe patient, based on the health information including one or more ofthe plurality of open care gaps in the open state; modifying, for atleast one of the open care gaps, the first value to be a second valuereflecting an artificially closed status of the care gap in the healthinformation; determining, using the trained predictor model with thesecond value as one of the input variables instead of the first value, asecond risk score for the patient; determining a predicted riskreduction score based on the first risk score and the second risk score;determining a patient classification based on the predicted riskreduction score, the patient classification comprising both a riskcomponent related to the first risk score and a benefit componentrelated to the second risk score; and outputting a report based on atleast one of the first risk score, the second risk score, or thepredicted risk reduction score.
 2. The method of claim 1, furthercomprising: prior to determining the first risk score, removingredundant health information and removing physiologically impossiblehealth information.
 3. The method of claim 1, further comprising: priorto determining the first risk score, imputing missing health informationusing one or more of linear interpolation from related healthinformation or robust multivariate imputation by chained equations. 4.The method of claim 1, further comprising: prior to determining thefirst risk score, discarding health information for which at least athreshold number of samples is missing.
 5. The method of claim 1,wherein the predictor model is a linear model.
 6. The method of claim 5,wherein the linear model is a linear logistic regression model.
 7. Themethod of claim 1, wherein the predictor model is a non-linear model. 8.The method of claim 7, wherein the non-linear model is one of randomforest or XGBoost.
 9. The method of claim 1, wherein the at least oneartificially closed care gap comprises a plurality of artificiallyclosed care gaps.
 10. The method of claim 1, wherein a split-by-yearprocedure applied to each trained predictor model of the plurality oftrained predictor models is used to determine which model is the bestmodel.
 11. The method of claim 10, wherein the best model is retrainedusing an optimal set of hyper-parameters.
 12. The method of claim 1,wherein the step of determining a patient classification comprisescomparing the predicted risk reduction score against predicted riskreduction scores of a plurality of other patients.
 13. The method ofclaim 12, wherein the step of determining a patient classificationfurther comprises ranking the patient relative to the plurality of otherpatients.
 14. The method of claim 1, wherein the patient is part of aheart failure population of patients.
 15. The method of claim 1, whereinthe patient is part of a population of at least one of diabetes,pulmonary disease, renal disease, rheumatologic disorders,musculoskeletal conditions, or endocrinopathies patients.
 16. The methodof claim 1, wherein the first risk score, the second risk score, and thepredicted risk reduction score relate to the clinical endpoint occurringwithin a predetermined period of time.
 17. The method of claim 16,wherein the clinical endpoint is mortality of the patient.
 18. Themethod of claim 16, wherein the predetermined period of time is 1 year.19. The method of claim 16, wherein the at least one artificially closedcare gap has a positive relationship with respect to the eventoccurrence.
 20. The method of claim 1, wherein the patientclassification includes evaluating the predicted risk reduction scorerelative to a number of patients needed to treat.
 21. The method ofclaim 1, wherein the report includes treatment recommendations for thepatient.
 22. The method of claim 21, wherein the treatmentrecommendations include palliative care.
 23. The method of claim 1,further comprising: allocating resources to the patient based on thepatient classification.
 24. The method of claim 1, wherein the trainedpredictor model is used in the step of determining a patientclassification based on the predicted risk reduction score.
 25. A methodfor providing treatment recommendations for a patient to a physician,the method comprising: receiving health information associated with thepatient, the health information comprising a plurality of inputvariables for a trained predictor model, at least some of the pluralityof input variables associated with a respective first value reflecting astate thereof, the health information including data derived from one ormore of echocardiogram or electrocardiogram signal data, wherein thetrained predictor model is trained on a dataset derived from EHR recordsof a patient training dataset; identifying a plurality of open care gapswithin the health information, each open care gap comprising an inputvariable of the plurality of input variables for which the respectivefirst value corresponds to an open state; determining, using the trainedpredictor model, a first risk score relating to a clinical endpoint ofthe patient, based on the health information including one or more ofthe plurality of open care gaps in the open state modifying, for atleast one of the open care gaps, the first value to be a second valuereflecting an artificially closed status of the care gap in the healthinformation; determining, using the trained predictor model with thesecond value as one of the input variables instead of the first value, asecond risk score for the patient determining a predicted risk reductionscore based on the first risk score and the second risk score;determining a patient classification based on the predicted riskreduction score, the patient classification comprising both a riskcomponent related to the first risk score and a benefit componentrelated to the second risk score; and outputting a report based on atleast one of the first risk score, the second risk score, or thepredicted risk reduction score.
 26. A method for providing treatmentrecommendations for a patient to a physician, the method comprising:receiving health information associated with the patient, the healthinformation comprising a plurality of input variables for a trainedpredictor model, at least one of the plurality of input variablesassociated with a respective first value reflecting a state thereof, thehealth information including data derived from one or more ofechocardiogram or electrocardiogram signal data; identifying a pluralityof open care gaps within the health information, each open care gapcomprising an input variable of the plurality of input variables forwhich the respective first value corresponds to an open state;determining, using the trained predictor model, a first risk scorerelating to a clinical endpoint of the patient based on the healthinformation including one or more of the plurality of open care gaps inthe open state, supplementing, by the trained predictor model, thehealth information to artificially change the first value to be a secondvalue reflecting an artificially closed status of the care gap in thehealth information; determining, using the trained predictor model withthe second value as one of the input variables instead of the firstvalue, a second risk score for the patient; determining a predicted riskreduction score based on the first risk score and the second risk score;determining a patient classification based on the predicted riskreduction score, the patient classification comprising both a riskcomponent related to the first risk score and a benefit componentrelated to the second risk score; and outputting a report based on atleast one of the first risk score, the second risk score, or thepredicted risk reduction score.